LIBRARY 

UNIVl  ...m  o^ 

CALIFORNIA 

SAN  DIEGO 


A    STATISTICAL    STUDY    OF 
LITERARY    MERIT 

With   Remarks    on    Some   New 
Phases  of  the   Method 


FREDERIC   LYMAN   WELLS.   PH.D. 

Palholojfioal  PN.vcliolofjist  in  tlio  .McL.oaii  HoKpital,  Waverley,  Mass. 
Formerly  I>ecturer  in  Barnard  College,  Colnnibla  University 


ARCHIVES   OF  PSYCHOLOGY 

EDITED  Br  R.  S.  WOODWOKTH 


NO.    7,   AUGUST,    1907 


Colninbia  University  Contribntions  to  PbilONOpby  and  Psycbologry, 

Vol.  XVI,  Wo.  3 


NEW  YORK 
THE    SCIENC 


:NC:£^R^S 


e 


CONTENTS 

PAGE. 

I.    The  Order,   Positions  and  Probable   Errors  of  Ten  Leading 

American  Authors 5 

II.    Quality  Analysis 20 

III.    On  the  Validity  of  Individual  Judgment  as  Measured  by  De- 

PiVRTURE    FROM    AN  AVERAGE 25 


I.     THE  ORDER,  POSITIONS,  AND  PROBABLE  ERRORS  OF 
TEN  LEADING  AMERICAN  AUTHORS 

The  practical  value  of  the  statistical  method  in  the  measurement 
of  a  mental  trait  rests  upon  the  hypothesis  that  such  value  of  this 
trait  as  is  worth  measuring  in  any  individual  is  significant  for  a  cer- 
tain group  of  persons  as  it  impresses  itself  upon  that  group,  and  only 
in  so  far  significant  as  it  thus  impresses  itself.  This  is  what  the  method 
measures.  Unrecognized  merit  may  exist,  but  it  is  also  likely  to  1^ 
inefficient  merit,  which  is  not  merit  at  all  in  any  legitimate  sense  of 
the  term.  We  must  finally  assume  efficiency  to  be  in  proportion  to 
its  influence.  This  would  work  injustice  only  where  such  influences 
were  unaccounted  for,  or  accounted  for  to  the  wrong  source,  and  in 
such  determinations  as  these  this  factor  is  certainly,  if  not  indeed  al- 
ways, negligible.  The  measure  of  influence  is  the  ultimate  criterion 
of  efficiency. 

While  the  data  of  the  method  are  based  upon  introspection,  yet 
they  are  dealt  with  in  such  a  wholly  objective  way  as  at  least  to  meas- 
ure, if  not  indeed  to  largely  remove,  the  invalidities  usually  traceable 
to  this  source.  Just  as  the  biologist  cannot  make  a  certain  measure- 
ment on  all  individuals  of  a  given  species,  so  here  we  cannot  deter- 
mine the  effect  of  our  objects  on  all  the  community.  We  need  not, 
however,  select  so  much  at  random  as  is  usually  advisable  for  the 
biologist,  but  we  can  select  those  individuals  whose  judgments  are 
the  least  likely  to  vary,  that  is,  those  best  informed  on  the  subject, 
just  as  the  biologist  would  select  as  assistants  those  individuals  who 
gave  him  the  smallest  variations  in  measuring  the  same  object.  We 
might  also  regard  the  judgment  of  each  grader  as  a  new  measurement 
made  with  the  same  instrument.  In  the  absence  of  constant  error, 
we  suppose  those  measurements  the  most  accurate  which  vary  from 
each  other  least.  We  should  find  that  persons  who  had  never  heard 
of  our  10  American  authors  would  grade  them  almost  by  pure  chance 
and  that  persons  of  limited  knowledge  in  this  respect  would  vary  a 
great  deal,  but  when  we  come  to  those  who  have  made  a  special  study 
of  this  group  there  is  but  little  variation,  and  it  is  their  judgment 
that  we  therefore  regard  as  the  most  valid.  As  we  ascend  the  scale, 
constant  deviations,  mainly  of  a  chronological  and  geographical 
nature,  are  introduced,  and  this  precludes  determinations  of  absolute 
validity.  It  is  not  these  that  would  be  of  most  use,  however,  but 
the  knowledge  of  how  the  series  of  graded  objects  has  influenced  a 
certain  particular  group.  From  this  point  of  view  the  method  is  as 
much  a  measure  of  the  judges  as  of  the  judged. 


6 


A  STATISTICAL  STUDY  OF  LITERARY  MERIT 


In  these  experiments  we  get  a  direct  measure  of  the  relative  ex- 
tent to  which  the  authors  have  impressed  themselves  upon  the  group 
which  we  are  studying.  In  so  far  as  this  is  a  representative  group, 
we  get  a  measure  of  the  extent  to  which  they  have  influenced  the  com- 
munity represented,  and  a  determination  based,  from  this  view-point, 
upon  entirely  objective  facts. 

The  writer's  first  experiment  by  this  method  along  the  lines  of 
literary  criticism  dealt  with  short  compositions  by  a  single  author, 
the  arrangements  being  made  by  40  women  undergraduates.  Ten 
stories  by  Edgar  Allan  Poe  were  graded  in  order  of  preference,  the 
order,  positions,  and  p.e.'s,  together  with  the  graphic  representation 
according  to  the  scheme  devised  by  Cattell,*  being  given  below. 


Order. 

Pos.           P. 

E. 

The  Fall  of  the  House  of  Usher. 

3-6 

26 

The  Murders  in  the  Rue  Morgue. 

4- 

35 

Ligeia. 

4.1 

22 

The  Purloined  Letter. 

4.6 

53 

William  Wilson. 

5.1 

24 

The  Telltale  Heart. 

5.8 

3 

The  Cask  of  Amontillado. 

6. 

38 

Metzengei  stein. 

6.6 

26 

Loss  of  Breath. 

7-1 

3 

Le  Due  de  L' Omelette. 

7-7 

32 

Average  Difference  in  Position 

.46    Av 

31 

On  account  of  the  limited  training  of  the  graders  the  m.v.'s  are 
considerable  compared  to  those  to  be  subsequently  discussed.  The 
differences  in  position  are  also  much  smaller.  "Working  by  the  method 
of  %  of  like  signs  it  was  not  possible  to  discover  any  correlations  m 
proferclic-e,  positive  or  negative,  that  might  not  as  well  be  ascribed 
to  pure  chance.  This  seems  rather  surprising,  as  one  would  naturally 
have  expected  relative  preferences  to  be  the  same  within  types  of 
stories,  that  is,  one  who  disliked  Loss  of  Breath  should  also  dislike 
Le  Diic  de  L'Omelette.  But  such  slight  relationships  as  did  appear 
seemed  to  be  rather  between  stories  relatively  unrelated  by  ordinary 
critical  standards,  as  positive  between  Loss  of  Breath  and  William 
Wilson,  negative  between  The  Purloined  Letter  and  The  Cask  of  Amon- 
tillado, etc. 

These  results  appeared  to  indicate  that  the  standards  of  literary 
^jriticisra  erected  by  accepted  critical  scholarship  would  bear  experi- 

*  Science,  N.  S.,  24,  658,  699,  732,  1906. 


ORDER,  POSITIONS,  ETC.  7 

mental  examination.  Aside  from  the  intrinsic  interest  of  determin- 
ing relative  positions  in  the  group  tested,  it  seemed  desirable  to  analyze 
so  far  as  possible  the  precise  standards  upon  which  such  judgments 
were  based.  Accordingly  the  experiment  whose  results  form  the 
raison  d'etre  of  the  present  study  was  devised.  It  is  not,  however, 
to  be  antici})ated  that  the  introduction  of  a  scientific  method  into 
this  field  should  contribute  markedly  to  the  principles  of  accepted 
critical  procedure;  the  main  function  of  literary  criticism  having 
hitherto  been  to  serve  rather  as  a  convenient  vehicle  for  individual 
expression  than  for  the  empirical  determination  of  actual  literary 
relationships. 

Ten  American  imaginative  writers  were  selected  for  study,  these 
being  presented  in  alphabetical  order,  Bryant,  Cooper,  Emerson, 
Hawthorne,  Holmes,  Irving,  Longfellow,  Lowell,  Poe,  Thoreau. 
They  are  presumably  all  in  the  first  15  of  their  class.  These  were 
graded  first  in  respect  to  general  literary  merit.  They  were  then 
graded  in  respect  to  their  possession  of  ten  literary  qualities.  These, 
also  in  alphabetical  order,  and  with  the  abbreviations  by  which  they 
will  subsequently  be  designated,  were  Charm  (Ch),  Clearness  (CI), 
Euphony  (Eu),  Finish  (Fi),  Force  (Fo),  Imagination  (Im),  Origi- 
nality (Or),  Proportion  (Pr),  Sympathy  (Sy),  Wholesomeness  (^Vh). 
These  lists  were  not  determined  by  any  standard  method  but  by  a 
literary  critic  in  ordinary  consultation  with  the  writer.  The  terms 
are  in  the  main  technical  terms  of  literary  criticism  and  there  seems 
to  have  been  no  great  difficulty  about  their  interpretation.  The 
grading  was  done  at  a  meeting  of  the  English  Graduate  Club  at  Colum- 
bia University,  the  work  occupying  from  35  minutes  to  1  hour.  One 
of  the  graders  was  the  critic  above  mentioned,  the  remainder  belong- 
ing, with  2  or  3  exceptions,  to  the  graduate  student  group.  There 
was  a  remarkably  small  amount  of  invalid  data,  principally  confined 
to  such  lapses  as  grading  the  same  author  3rd  and  then  again  7th. 
The  present  results  are  derived  from  20  records. 

Of  course  in  so  large  a  number  of  separate  distributions  as  that  un- 
der consideration  (110),  the  probable  incidence  of  certain  forms  by 
pure  chance  is  not  inconsiderable.  While  in  general  they  approxi- 
mate the  normal  distribution  as  closely  as  could  be  expected  in  the 
limited  number  of  judgments,  yet  it  may  be  worth  while  to  call  atten- 
tion, with  special  reference  to  species,  to  some  of  the  more  marked 
deviations  from  the  normal,  where  the  factor  of  chance,  which,  of 
coiu-se,  is  itself  always  measurable,  does  not  seem  to  play  a  promi- 
nent part. 

This  is  perhaps  the  phase  of  the  results  most  interesting  to  stu- 
dents of  literature.  For  example,  the  fact  that  VII  (Bryant)  has  a 
distribution  of  such  marked  bimodality  as  to  be  practically    with- 


8  A  STATISTICAL  STUDY  OF  LITERARY  MERIT 

out  the  range  of  chance  deviation  from  the  normal,  is  perhaps  not 
without  critical  interest.  It  has  been  suggested  that  these  two  groups 
might  have  a  certain  geographical  distribution,  the  relatively  higher 
grades  coming  from  New  England  and  neighboring  states.  It  is  now 
impracticable  to  verify  this  supposition,  but  there  is  nothing  inher- 
ently improbable  about  it.  and  such  theories  are,  of  course,  experi- 
mentally verifiable.  I  am  rather  distrustful,  however,  of  the  value 
of  explanation  for  its  own  sake  and  representing  a  personal  opinion. 
We  shall  perhaps  do  well  to  remember  that  we  know  just  as  good 
reasons  for  many  things  that  are  not  so  as  for  things  that  are,  and 
when  the  history  of  our  present  thought  is  written  it  will  probably 
be  found  that  we  have  explained  to  our  complete  satisfaction  quite 
as  many  of  the  former  as  the  latter. 

There  is  little  ground  for  supposing  different  species  in  the  re- 
mainder of  the  general  merit  grades  except  perhaps  in  the  case  of 
VIII  (Thoreau),  whose  grades  fall  with  almost  equal  frequency 
among  the  last  5  positions.  The  three  most  markedly  bimodal  dis- 
tributions m  the  quality  grades  are  those  of  II  (Poe)  for  Charm,  and 
I  (Hawthorne)  for  Clearness  and  Sympathy.  In  17  cases  the  same 
author  receives  grades  in  first  and  last  place,  though  in  only  2  cases 
is  there  a  grade  in  every  place,  namely,  in  IV  (Lowell)  for  Sympathy 
and  X  (Cooper)  for  Clearness.  The  most  variable  distribution  is 
that  of  III  (Emerson)  for  Proportion  with  a  p.e.  of  .61,  and  the  least 
variable  are  those  of  II  for  Imagination  and  Originality  with  p.e.'s 
of  .11  each.  There  are  naturally  many  distributions  that  on  their 
face  are  bimodal,  but  the  probability  of  their  occurrence  by  pure  chance 
is  too  great  to  warrant  their  acceptance  as  evidences  of  species  in 
the  judgments.  On  the  whole,  the  opinions  seem  to  concentrate 
about  a  common  centre  rather  than  to  form  groups. 

If  the  distributions  were  governed  by  pure  chance,  they  would  . 
always  approximate  to  2  grades  in  each  place.  As  the  frequencies 
are  not  governed  by  pure  chance,  but  presumably  by  the  probability 
distribution  about  a  mode,  we  can  roughly  determine  to  what  ex- 
tent the  variability  we  obtain  is  a  true  variability  for  this  class  of  judg- 
ments. For  example,  in  the  40  judgments  of  Poe's  stories,  it  was 
found  that  the  results  from  20  random  selections  differed  but  little 
from  the  results  of  the  40.  There  would  thus  be  reason  to  believe 
that  the  variability  found  in  the  40  judgments  was  representative 
of  the  amount  of  variability  that  we  might  expect  to  find  in  dealing 
with  judgments  of  this  sort.  It  has  been  suggested  that  in  this  method 
at  least,  the  reliability  increases  much  more  slowly  than  as  the  square 
root  of  the  numl^er  of  cases,  and  may  be  more  accurately  represented 
by  the  mean  variation  itself. 

If  the  factor  of  memory  might  only  be  overcome,  it  would  be 


ORDER,  POSITIONS,  ETC.  9 

well  wortli  while  to  compare  with  the  variability  of  nuiiiy  individuals 
the  variability  of  a  single  individual  from  the  averajje  of  his  own 
judgments.  This  was  done  l)y  Cattell  for  a  consideral^le  numlK-r  of 
psychologists.  We  shouhi  then  have  a  measure  of  constancy  in  judg- 
ment that  would  have  a  not  uninteresting  psychological  bearing.  A 
single  judgment  is  subject  not  only  to  error  from  the  average  judgment 
of  other  individuals,  but  from  the  average  judgment  of  the  individual 
himself.  Large  and  small  m.v.'s  may  be  the  product  of  variations 
along  either  of  these  lines.  We  are  all  probably  very  much  surer  of 
our  relative  preferences  for  lobster  Newberg  and  fried  oysters  than  of 
our  preferences  for  Emerson  and  Hawthorne;  yet  these  very  differ- 
ences in  taste  might  produce  as  large  an  m.v.  in  one  case  as  in  the 
other. 

For  some  purposes  of  analysis  the  median  has    seemed  a  better] 
measure  than  the  average.     It  was  somewhat  discredited  in  the  results  , 
of  Cattell,  but  is  of  more  value  here  on  account  of  the  larger  number 
of  measures.    The  average  is  here  also  relatively  less  valid  because' 
the  number  of  possible  positions  is  limited  to  ten,  whereas  it  was  there 
in  the  negative  diiection  practically  unlimited.     In  the  present  re- 
sults there  is  almost  no  distribution  in  which  the  author  does  not  re- 
ceive  a  grade  in  either  first  or  last  place,  and  when  the  grades  are 
banked  up  against  first  or  last  place,  the  average  is  obviously  too  low 
or  too  high,  probably  more  so  than  the  median.     However,  it  is  of 
no  particular  consequence  which  we  use  so  far  as  order  is  concerned, 
for  the  two  orders  are  almost  identical,  the  divergences  that  occur 
being  well  within  the  limits  of  chance  variation. 

The  accompanying  tables  give  the  main  results  of  the  experiment 
in  the  median  and  average  order  and  position  of  the  authors  in  general 
merit  and  the  equalities. 

In  general  merit  the  writers  fall  into  three  groups,  separated 
by  considerable  distances,  three  at  the  top,  three  in  the  middle,  and 
four  at  the  bottom.  Between  the  three  at  the  top  there  is  little  differ- 
ence to  speak  of,  between  I  and  II  practically  none  at  all.  The  median 
of  II  is  considerably  higher  than  that  of  I,  and  it  is  very  possible  that 
his  true  position  is  higher  than  I.  Such  constant  error  as  might  result 
from  prejudice  would  perhaps  operate  more  against  II.  P^ach  has 
six  grades  in  first  place,  and  none  in  last.  It  is  quite  anomalous  that 
the  differences  should  be  greater  in  the  middle  group  than  at  the 
ends;  although  the  p.e.'s  are  not  of  the  smallest  they  fail  to  overlap 
at  all;  the  chances  are  over  16-1  that  the  order  given  is  correct.  The 
narrow  mathematical  limits  of  variability  might  account  in  a  measure 
for  the  small  p.e.'s  at  the  ends,  and  perhaps  also  for  the  small  differ- 
ences in  position,  which  are  equally  striking;  but  only  in  a  small  meas- 
ure, for  this  condition  does  not  obtain  in  the  quality  grades,  nor  in 


10 


A  STATISTICAL  STUDY  OF  LITERARY  MERIT 


MEDIANS. 

G.M. 

Ch. 

CI. 

Eu. 

Fi. 

Fo. 

Im. 

Or. 

Pr. 

Sy. 

Wh. 

M.ofM. 

Hawthorne        I . 
Poe                   II. 
Emerson         III. 
Lowell             IV. 
Longfellow       \' . 
Irving              VI . 
Bryant           VII. 
Thoreau       VIII . 
Holmes           IX. 
Cooper              X . 

2 
2 
2 
4 
5 
6 
6 
8 
8 
8 

5 

2 

7 

5 

5 
8 

2 

4 
6 

4 
5 
2 
8 
5 
4 
8 

2 
7 
7 
5 

I 
5 
7 
3 
9 

5 

6 

9 
6 

3 
2 
6 
7 
4 
5 

2 
6 

5 
5 
5 

I 
3 

3 

I 

7 
3 
3 
4 
5 
7 
7 
9 

2 
5 
7 
8 

5 
5 
3 
5 

7 

I 
2 

7 
4 
3 
4 
5 
7 
7 
9 

5 

8 
9 
7 
7 
8 

5 
7 

4 

2 
I 
4 
7 
8 

5 
4 
7 
6 

5 
3 
3 
2 

7 
7 
7 
2 

7 
7 

2. 

I . 

5-1 

6.1 

6. 

6. 

7- 

7-5 

8.3 

2.7 

I 
3 
6 
8 
6 
7 
4 
7 
5 

7 
2 

2 
I 
3 
3 

5 
5 

2 

2 

-7 
/ 

4 
4 
3 
7 
8 
6 
8 

7 
5 

I 
8 

9 

8 

3 
9 
5 
4 
3 
3 
6 

7 
5 
7 

7 
3 
5 

5 

5 

7 

7 

6.9 
10 

2.5 

4-5 

3-2 

4.1 

5- 
5-7 
5-1 
7.2 

2.9 

2.  I 

6.1 

4-5 

4- 

4-3 

6.2 

7.2 

7.2 

7-5 

Medians        5 

5 

4 

7 

5 

5 

5 

5 

8 

5 

I 

6. 

5 

9 

5 

5 

5 

2 

4 

8 

5-3 

Median  Orders.    Positions  Displaced  from  Average  Order  are  Given  in  Italics. 

G.  M. 

Ch. 

CI. 

Eu. 

Fi. 

Fo. 

Im. 

Or. 

Pr. 

Sy. 

Wh. 

//. 

VI. 

VI. 

II. 

I. 

III. 

II. 

II. 

II. 

V. 

III. 

I. 

I. 

V. 

I. 

II. 

II. 

I. 

I. 

I. 

VI. 

V. 

III. 

IX. 

IX. 

IV. 

V. 

I. 

X. 

III. 

VI. 

I. 

VI. 

IV. 

IV. 

X. 

V. 

VI. 

VIII. 

III. 

VIII. 

IV. 

IV. 

IV. 

V. 

//. 

I. 

VI. 

IV. 

IV. 

V. 

A". 

V. 

IX. 

VII. 

VI. 

V. 

VII. 

VII. 

VII. 

VII. 

VI. 

IV. 

IX. 

III. 

IX. 

VII. 

VIII. 

//. 

VIII. 

III. 

X. 

IV. 

VI. 

VII. 

VII. 

VIII. 

VIII. 

III. 

IV. 

IX. 

IX. 

IX. 

VII. 

VII. 

III. 

X. 

I. 

IX. 

VII. 

VIII. 

III. 

VIII. 

V. 

VIII. 

IX. 

VIII. 

VIII. 

X. 

X. 

X. 

III. 

X. 

X. 

VI. 

IX 

V. 

X. 

II. 

II. 

AVERAGES. 

G.M. 

Ch. 

Cl. 

Eu. 

Fi. 

Fo. 

Im. 

Or. 

Pr. 

Sy. 

Wh. 

Av.  of  Av. 

I. 

II. 
III. 

IV. 

V. 

VI. 

VII. 

VIII. 

IX. 

X. 

A.  D.  p. 

2 
2 
2 

4 
5 
5 
7 
7 
8 
8 

5 
6 

9 
4 

I 

7 
I 

9 

I 

4 

3 
4 
6 

5 
5 
2 

7 
6 

5 
8 

2 

I 
7 

5 
9 

7 
5 

I 

5 

5 
6 
8 
5 
3 
3 
5 
6 

5 
5 

I 
6 
3 
5 
5 

5 
7 
2 

I 

3 

I 

7 
4 
3 
4 
5 
7 
7 
9 

4 
7 
6 
I 

7 
8 
6 
2 

8 

I 

2 
2 
6 

4 
4 
4 
5 
7 
7 
9 

I 
2 

7 
8 

5 
6 
6 
5 
3 
4 

4 
3 

I 

4 
7 
8 
6 

4 
6 
6 

5 
4 
7 
8 

5 

2 

3 
8 
6 

2.4 

1-4 
5-8 
6.1 
6.2 
6.1 
6.8 

7.3 
8. 

4-7 

2 

I 

3 
6 

7 
6 
6 
4 
7 
6 

9 
5 
8 
6 
8 
2 

7 
8 

9 

4 

3 
2 
6 
5 
4 
4 
5 
7 
6 
8 

I 
9 
5 

3 
2 

9 

4 
9 
5 

3 
8 

5 
4 
3 
3 

7 
8 

5 

7 

5 
4 
6 

4 
2 
8 
I 

I 
6 

6 
9 

3 
4 
4 
4 
4 
5 
5 
6 

4 
5 
7 

9 
9 
6 

4 
4 

3 
4 
5 
5 
5 
4 
6 
6 
6 
7 

7 
2 
6 

9 
2 

7 
5 

2 

65 

62 

59 

82 

81 

72 

•^^1 

71 

62 

58 

51 

•39 

ORDER,  POSITIONS,  ETC. 


11 


PROBABLE  ERRORS. 

G.M. 

Ch. 

CI. 

Eu. 

Fi. 

Fo. 

Im. 

Or. 

Pr. 

Sy. 

Wh. 

Av. 

I. 

II. 

III. 

IV. 

V. 

VI. 

VII. 

VIII. 

IX. 

X. 

Av. 

21 

25 

37 
35 
25 
31 

35 
37 
21 

33 

■37 
■47 
■31 
■31 
■45 
■35 
.29 
•38 
■45 
■23 

56 

41 

48 
56 

29 
29 

31 

38 
38 

47 

29 
19 
33 
27 
35 
43 
35 
47 
27 
17 

33 
21 

38 
41 
27 
31 
35 
31 
27 
23 

29 

41 
17 
37 
31 
25 
31 
38 
31 
48 

•17 
.  1 1 

■48 
■31 
•43 
■35 
•33 
■33 
■27 
■45 

23 
1 1 

43 
31 
35 
33 
29 
45 
33 
39 

•31 

■37 
.61 
■52 
■31 
•33 
•33 
■33 
.29 

■25 

33 
31 
51 
39 
38 
31 
47 
35 
37 
19 

37 
15 

52 
27 
35 
41 
43 
48 

37 
41 

32 

27 
42 
37 
35 
33 
34 
38 
33 
32 

3 

-36 

41 

31 

31 

33 

•32 

32 

•36 

36 

38 

343 

i 

- 

1 

_ii 

other  relative  position  work  that  has  been  done  with  even  smaller 
series  than  10.  Between  the  positions  of  VI  and  VII  is  another  long 
step,  1.4  between  positions,  .8  between  limits  of  p.e.'s,  and  VII 
again  fails  to  overlap  the  p.e.  of  VIII.  From  here  until  X's  posi- 
tion at  8.4  the  steps  are  about  equal. 

It  is  thus  seen  that  we  have  no  man  who  is  so  distinctly  at  the 
head  of  American  writers  as  one  is  found  among  contemporary  Astrono- 
mers, Psychologists  and  Pathologists.  It  is  perhaps  a  fair  inference 
that  enlargement  of  a  group  may  decrease  differences  at  the  top  by 
bringing  more  of  the  leaders  into  conflict.  There  is  no  doubt  that 
a  certain  department  of  American  letters  could  have  been  found 
in  which  III  would  have  reigned  supreme,  and  the  differences  between 
I  and  II  could  have  "been  much  increased,  in  either  direction,  by  nar- 
rowing the  field  of  literary  work  to  be  considered.  It  is  beyond  dis- 
pute that  there  would  be  more  disagreement  about  the  order  and  less 
about  the  identity  of  the  five  greatest  poets  of  the  world  than  the 
five  greatest  poets  of  France.  Such  a  condition  is  probably  to  be  ex- 
pected in  all  walks  of  life.  There  is  a  limit  to  the  realization  of  human 
powers  fixed  by  opportunity  and  other  environmental  factors.  "Es 
wird  dafiir  gesorgt,"  says  the  German  proverb,  "dass  die  Baume 
nicht  in  den  Himmel  wachsen."  If  we  artificially  limited  to  140 
ft.  the  height  of  a  tree  ordinarily  growing  to  150  ft.,  we  should  find 
more  trees  at  140  than  at  135.  It  acts  in  the  same  way  as  any  other 
limitation  of  a  normal  distribution,  crowding  the  extreme  cases  to- 
gether. This  is  probably  a  reasonable  alternative  to  the  supposition 
of  genius  as  a  separate  group. 

Though  the  peculiar  conditions  noted  above  do  not  generally 
obtain  in  the  qualities,  these  present  certain  other  points  of  interest. 
In  Charm  there  is  a  group  slightly  above  the  middle  position,  the  in- 
creases and  decreases  from   which   show  nothing  anomalous.     The 


12 


A  STATISTICAL  STUDY  OF  LITERARY  MERIT 


G.  M. 

Ch. 

CI. 

Eu. 

u 

O 

OS 

5 

'in 

s, 

1^ 

■a 
O 

s 

o 

« 

3.^ 

u 

0 

0 
'5) 

CI 

•0 
0 

01 

c 

0 

I. 

II. 

III. 

IV. 

V. 

VI. 

VII. 

VIII. 

IX. 

X. 

2 
2 
2 

4 
5 
5 
7 
7 
8 
8 

5 
6 

9 

4 
1 

7 

I 

9 

I 

4 

21 
25 

37 
35 

25 

31 
35 
37 

21 

33 

VI. 

I. 

II. 

IV. 

IX. 

V. 

VIII. 

III. 

VII. 
X. 

2 

3 
4 
5 
5 
5 
6 
6 

7 
8 

9 

2 
I 

I 

5 
5 
7 
7 
5 

35 
37 
47 
31 
45 
45 
38 
31 

29 

23 

VI. 
V. 
X. 

I. 

IX. 

VII. 

IV. 

II. 

VIII. 

III. 

3 
3 
5 
5 
5 
5 
5 
6 
6 
8 

5 
I 
I 
2 
5 
5 
6 

7 
3 

29 
29 

47 
56 
38 
31 
56 
41 
38 
48 

II. 
I. 

V. 

IV. 

VI. 

VII. 

VIII. 

III. 

IX. 
X. 

I 

3 
3 
4 
4 
5 
7 
7 
7 
9 

7 
4 
7 
I 
8 
6 
2 
6 
8 
I 

19 
29 

35 
27 
43 
35 
47 
33 
27 
17 

Fi.                    1                     Fo. 

Im. 

Or. 

u 
V 

■E 
0 

c 
0 

■55 

W 
Pi' 

u 

11 

0 

5 

1 

t4 

u 

<v 

■a 

u 

0 

s 

0 
'm 

0 

Ph 

u 

0 

0 

■35 

0 

I. 

II. 

V. 

VI. 

IV. 

VII. 

III. 

IX. 

VIII. 

X. 

2 

2 

4 
4 
4 
5 
6 

7 
7 
9 

I 
2 

5 
6 
8 
6 
7 
3 
5 
4 

33 
21 

27 
31 
41 
35 
38 
27 

31 

23 

III. 

II. 

I. 

VIII. 

IV. 

X. 

VII. 

IX. 

V. 

VI. 

1-7 
3-4 
4-5 
4.8 
4.8 
6. 

6.3 
6.6 

7-5 
8.2 

•17 
•41 
.29 

■38 

•37 
.48 
•31 
■31 
•31 
■25 

II. 
I. 

X. 

III. 

VI. 

IV. 

V. 

VII. 

VIII. 

IX. 

1-4 

2.4 

4-7 
5-8 
6.1 
6.1 
6.2 
6.8 

7-3 

8. 

II 

17 
45 
48 

35 
31 
43 
33 
33 
27 

II. 

I. 

III. 

VIII. 

VI. 

X. 

IV. 

VII. 

V. 

IX. 

1-5 
2.9 
3-8 
4.8 
6.2 
6.4 
6.6 
6.7 
7.8 
7-9 

II 

23 

43 
45 
33 
45 
31 
29 

35 

27 

Pr. 

Sy. 

Wh. 

Av.  Resp.  Pos. 

V 

•0 
0 

tn 

a 
1 

t4 

U 
V 

■E 
0 

c 
0 

W 
Ph' 

•0 

0 

in 

a 
0 

1 

1 

II. 

I. 

VI. 

V. 

IV. 

VII. 

III. 

IX. 

VIII. 

X. 

2 

3 
4 
4 
5 
5 
6 
6 

7 
8 

9 

I 
2 
3 

9 
5 
9 
4 
5 

37 
31 
33 
31 
52 
33 
61 
29 
33 
25 

V. 

I. 

VI. 

IV. 

IX. 

III. 

VII. 

X. 

VIII. 

II. 

3 
3 
3 
4 
5 
5 
7 
7 
8 
8 

2 

5 
8 

4 
I 
6 
I 
6 

4 

•38 

■33 

•31 

29 

•37 
•51 
•47 
•19 
■35 
•31 

III. 

V. 

IV. 

VI. 

VII. 

IX. 

VIII. 

X. 

I. 
II. 

3 
4 
4 
4 
4 
5 
5 
6 
6 
9 

7 

9 
9 

4 
6 

4 
4 
5 

•52 

■35 
■  27 
•43 
•41 
•37 
•48 
•  41 
•37 
•15 

2.4 
3^2 

42 
4.8 
52 

5^8 
6.4 
6.9 

7-4 
8.6 

25 

3 

36 

39 

4 

39 

34 

31 

33 

26 

ORDER,  POSITIONS,  ETC.  13 

author  in  first  place,  VI,  is  ordinarily  noted  for  his  Charm,  and  the 
fact  that  he  is  so  hard  pressed  by  I  may  mean  that  he  is  so  nf)ted  more 
than  he  deserves.  He  is  graded  rather  for  its  prominence  relative 
to  his  own  other  c[ualities.  Clearness  again  gives  us  two  positions  at  the 
top,  VI  and  V,  and  widely  separated  from  the  next  five,  who  fonn 
the  largest  single  group  in  the  results.  The  p.e.'s  are  unusually 
large.  It  is  also  peculiar  that  the  lowest  individual  in  the  quality 
III,  has  also  the  largest  p.e.  in  it,  the  only  case  among  the  qualities 
where  the  last  p.  e.  is  not  smaller  than  the  average- 
Euphony  has  one  of  the  widest  ranges  and  is  among  the  smallest 
p.e.'s.  The  leader,  II,  and  the  last,  X,  are  a  long  distance  from  any 
of  their  fellows,  while  the  remainder  fall  into  two  groups,  the  upper 
of  five  and  the  lower  of  three,  separated  by  an  interval  of  nearly  a 
place.  The  distribution  in  Finish  is  a  composite  of  those  in  Clear- 
ness and  Euphony,  there  being  two  leaders,  I  and  II,  and  a  distinct 
last  place,  X,  as  in  Clearness,  without  the  closely  packed  group  of  that 
quality.  Force  again  has  a  distinct  leader.  III,  but  the  remainder 
trail  behind  with  no  characteristic  variations  in  successive  distance. 
The  same  is  true  of  Imagination  except  that  there  are  two  leaders, 
II  and  I,  though  the  difference  between  them  is  itself  not  inconsidera- 
ble. There  is  a  marked  group  as  in  Clearness,  but  here  centered 
at  a  position  lower  than  the  average.  In  Originality  the  first  four 
positions,  II,  I,  III,  and  VIII,  are  established  well  beyond  the  limits 
of  p.e.  Then  comes  a  closely  packed  group  of  four,  and  separated 
from  these  by  an  interval  of  about  a  place  are  the  two  lowest  posi- 
tions. The  distribution  is  quite  similar  to  that  of  Imagination. 
Proportion  resembles  so  closely  the  distribution  of  Charm  that  the  same 
may  be  said  of  them  in  essential,  save  that  in  Proportion  there  is  not 
so  distinct  a  grouping.  In  Sympathy,  whose  range  is  also  of  the  small- 
est, the  p.e.'s  all  overlap  save  for  the  considerable  break  between 
6th  and  7th  positions.  In  Wholesomeness  the  first  nine  are  distribu- 
ted over  a  very  small  range,  and  the  tenth — II  in  general  merit — 
brings  up  the  rear  with  the  largest  difference  and  one  of  the  smallest 
p.e.'s  of  the  results.  The  final  figure  gives  the  average  position 
and  average  p.e.  of  each  place  in  the  above  qualities,  irrespective  of 
the  author  holding  it.  The  first  and  second  positions  are,  as  a  rule, 
determined  with  some  certainty,  as  is  also  the  last.  In  all  the  re- 
mainder the  p.e.'s  show  a  slight  and  very  constant  overlapping. 

The  p.e.'s  have  been  calculated  by  the  simple  formula  advocated 

.845  A.  D. 
in  Cattell's  Statistics  of  Americaji  Psychologists,  i.  e.,  p.e.  =      ^~zri 

They  are,  as  has  been  noted,  probably  smaller  than  is  representative 
of  the  actual  reliability  of  the  determinations.  It  will  be  noted,  how- 
ever, that  they  are  quite  consistently  larger  in  the  qualities  than  for 


14         A  STATISTICAL  STUDY  OF  LITERARY  MERIT 

general  merit.  This  may  be  taken  to  mean  that  we  really  differ 
less  in  personal  opinion  about  a  general  attribute  than  about  its  con- 
stituents, or  that  in  these  constituents  there  are  likely  to  be  smaller 
differences  than  in  general  attributes.  Judgment  of  general  merit 
may  be  more  variable  than  judgment  of  special  merit,  and  general 
merit  may  itself  be  more  variable  than  special  merit.  Under  the  pres- 
ent circumstances,  however,  we  seem  to  have  a  fairly  complete  list 
of  qualities,  of  which,  on  the  former  supposition,  some  should  be  more 
variable,  some  less  variable  than  their  total.  As  a  matter  of  fact, 
the  quality  grades  are  all  more  variable  than  those  in  general  merit, 
which  seems  to  point  to  the  latter  interpretation  as  the  more  valid 
one,  especially  when  we  consider  that  the  average  difference  in  con- 
secutive position  (A.D.P.)  is  in  but  five  cases  out  of  ten  greater  in 
the  qualities  than  in  general  merit.  The  fact  seems  to  be  that  the 
differences  are  more  variable  in  the  qualities;  the  median  difference 

Graphic  Representation  of  the  Average  Positions  and  P.E.s  Given  in  the  Last 

Preceding  Table. 


General  merit. 


Charm. 


Charness. 


Euphony. 


Finish. 


3  t  i'  fc  1  *?  9  '" 


ORDER,  POSITIONS,  ETC.  15 


Imagination. 


Originality. 


Proportion. 


Sympathy. 


Wholesomeness. 


Average  of  respec- 
tive positions. 


In  position  would  here  be  smaller.  The  extreme  p.e.'s  are  also  smaller 
in  the  qualities  than  in  general  merit.  In  certain  isolated  cases, 
as  that  of  II  for  Wholesomeness,  it  seems  that  judgment  is  surer  than 
for  general  merit,  but  it  may  also  go  farther  astray,  and  is  usually 
less  accurate  than  for  generalities. 

It  is  a  not  uncommon  observation  that  we  often  form  judgments 
for  which  we  cannot  give  satisfactory  reasons,  and  it  is  perhaps  not 
less  common  to  observe  that  these  judgments  are  about  as  likely  to 
be  correct  as  those  for  which  we  can.  To  this  empirical  generaliza- 
tion the  above  figures  seem  to  lend  experimental  support.  We  are 
more  accurate  in  our  opinions  than  in  our  reasons  for  them. 

The  p.e.'s  are  of  some  interest  in  themselves,  quite  apart  from 
the  positions  to  which  they  attach.  On  a  scale  of  .05  they  are  dis- 
tributed as  follows : 

.10    .15    .20    .25    .30    ,35     40    .45    .50    .55     60 
26         6      17      28      13      10        7        3        2         I     Av.  .34  m.v.   .079 


16         A  STATISTICAL  STUDY  OF  LITERARY  MERIT 

The  largest  and  smallest  p.e.'s  are,  as  has  been  noted,  those  of 
III  in  Proportion  and  II  in  Imagination  and  Originality.  The  dis- 
tribution is  skewed  toward  the  small  end,  indicating,  if  anything,  a 
sort  of  psychological  limit  in  variability,  just  as  we  assume  a  physio- 
logical limit  of  quickness  to  account  for  similar  distributions  in  reac- 
tion time.  This  would  probably  be  determined  by  the  individual's 
chance  variations  from  his  own  judgments.  Even  if  there  were  com- 
plete agreement  of  the  average  opinions  of  the  individuals  we  should 
not  get  p.e.'s  of  .0,  because  no  single  measure  would  give  us  this  aver- 
age. 

The  distribution  is  quite  regular,  with  no  surface  indication& 
of  species,  but  analysis  makes  it  rather  probable  that  they  exist. 
Each  of  the  p.e.'s  represents  roughly  the  accuracy  with  which  one  of 
the  authors  can  be  graded  in  one  of  the  qualities.  We  should  natur- 
ally expect  that  some  authors  would  be  more  accurately  graded  than 
others.  On  comparing  the  average  p.e.'s  of  the  authors'  quality 
grades  we  find  an  order  fairly  distinct,  though,  of  course,  itself  sub- 
ject to  a  large  p.e.  II  is  the  author  about  whom,  wherever  he  is  placed, 
there  seems  to  be  the  least  all-roimd  disagreement;  the  average  of 
his  p.e.'s  is  only  .27.  On  the  other  hand,  there  is  the  greatest  dis- 
cord about  his  neighbor,  III,  his  corresponding  figure  being  .42.  The 
complete  order  of  accuracy  in  w'hich  the  qualities  of  the  authors  are 
estimated  is  II,  I,  X,  IX,  VI,  VII,  V,  IV,  VIII,  III.  As  will  be  seen, 
this  order  bears  no  direct  relation  to  that  of  general  merit,  but  we  do 
have  a  logical  result  in  there  being  the  least  disagreement  about  those 
at  the  ends  of  the  list.  Such  a  fact  indicates  that  there  is  no  great 
difference  in  information  about  the  authors  as  related  to  their  posi- 
tions. As  it  is  fair  then  to  assume  that  we  know  nearly  as  much 
about  the  last  man  as  about  the  first,  we  probably  know  approx- 
imately as  much  about  those  in  the  middle,  their  higher  p.e.'s  being  due 
to  more  marked  differences  of  opinion  about  them. 

A  curious  sidelight  upon  this  situation  is  thrown  by  the  fact 
already  brought  out  in  the  last  of  the  diagrams  referred  to  on  p.  15, 
An  analogous  result  is  obtained  from  the  average  of  the  p.e.'s  taken 
in  respective  order.  That  is,  if  we  average  all  the  p.e.'s  of  the  first 
positions  in  the  qualities,  then  those  of  all  the  second  positions,  etc.,. 
we  obtain  a  quite  regular  increase  at  the  middle  and  decrease  at  the 
ends.  This  can  not  be  called  surprising  in  view  of  the  results  men- 
tioned in  the  preceding  paragraph,  but  it  is  hardly  what  should  have 
been  expected  a  priori.  It  is  as  though  we  had  in  these  authors 
stumbled  upon  a  range,  or  grouping  of  excellence  in  the  literary 
qualities.  The  artificial  limits  of  the  p.e.  do  not  seem  to  suffice  for 
the  facts.  This  is  a  result  contrary  to  that  given  in  the  positions  of 
general  merit,  in  which  differences  were  greatest  in  the  middle.     In 


ORDER,  POSITIONS,  ETC.  17 

respect  to  the  qualities,  the  authors  seem  to  form  a  group,  their  rela- 
tive positions,  of  course,  differing  widely  for  each  quality.  In  respect 
to  the  direct  general  merit  grades  they  can  be  considered  only  as 
part  of  a  group  or  as  three  sub-groups.  The  discrepancy  may  Ije  ac- 
counted for  by  the  wide  range  in  importance  of  the  qualities  and  the 
lack  of  correlation  between  them. 

In  the  same  way  as  with  the  authors,  we  should  also  expect  it  to 
be  possible  to  grade  certain  qualities  more  accurately  than  others. 
Comparing  the  averages  of  the  p.e.'s  for  the  qualities,  we  see  that 
this  is  to  some  extent  the  case,  though  the  range  is  not  so  large  as  with 
the  authors.  The  most  accurately  graded  of  the  qualities  has  an 
average  p.e.  of  .307,  the  least  one  of  .413,  the  order  being  Euphony, 
Finish,  Imagination,  Originality,  Force,  Proportion,  Sympathy, 
Charm,  Wholesomeness,  Clearness.  The  size  of  this  average  p.e. 
corresponds  generally  to  the  A.D.P.  of  the  authors  in  the  various 
qualities;  where  the  p.e.  is  smallest,  the  A.D.P.  is  greatest,  as  we  should 
expect.  It  would  seem  almost  tautological  to  say  that  the  accuracy 
with  which  differences  were  perceived  would  be  dependent  on  their 
size. 

However,  this  does  not  seem  to  be  necessarily  the  case,  as  is 
sho^vn  in  the  results  of  Cattell.  We  may  have  equal  differences  in 
position  with  miequal  p.e.'s,  and  equal  p.e.'s  attached  to  very  unequal 
differences  in  position.  Though  it  would  hardly  make  much  differ- 
ence in  the  upper  ten  positions,  the  cases  most  comparable  are  those 
in  which  an  equal  number  of  workers  are  considered.  Such  cases 
occur  between  Physics-Zoology,  Botany-Geology,  and  Astronomy- 
Psychology.  The  relations  of  the  p.e.  to  the  A.D.P.  are  in  these  cases 
as  follows: 

Phys.    Zool.  Bot.        Geol.  Ast.    Piych. 

A.D.P.  first  ten  positions  1.2        1.2  1.05       1.75  -8         -93 

Av.  p.e.  first  ten  positions        3.2       1.6         1.54       i-6  -6         -8 

The  size  of  the  p.e.  seems  to  a  certain  extent  independent  of 
the  differences  in  position.  The  A.D.P.  of  the  first  ten  botanists  is 
less  than  that  of  the  first  ten  geologists,  but  the  graders  of  the  geolo- 
gists are  slightly  less  reliable  than  those  of  the  botanists.  It  seems, 
on  the  one  hand,  that  individuals  may  differ  more,  yet  on  the  other 
hand  it  may  be  impossible  to  estimate  the  differences  with  so  great 
precision.  It  would  hardly  be  profitable  to  discuss  the  conditions 
of  such  a  relationship  save  upon  the  basis  of  empirical  analysis,  for 
which  the  small  ranges  obtained  in  the  present  study  hardly  afford 
suflricient  material.  The  variability  of  individual  gradings  might 
also  be  an  essential  factor.  It  is  evident,  however,  that  neither  figure 
alone  expresses  the  sum  total  of  the  differences.  Professor  Cattell 
has  employed  a  correction  for  the  range,  which  gives  the  various  p.e.'s 


18         A  STATISTICAL  STUDY  OF  LITERARY  MERIT 

more  strictly  comparable  values.  The  relationship  might  also  be 
expressed  in  tenns  of  the  ratio  of  the  p.e.  to  the  A.D.P.  This  would 
furnish  a  rough  index  of  the  adaptability  of  different  problems  to  meas- 
urement by  relative  position.  The  general  ratio  of  p.e.  to  A.D.P. 
in  the  present  determinations  is  about  1:2;  taking  the  p.e.  in  its 
literal  interpretation  this  would  mean  that,  by  and  large,  we  could 
measure  such  differences  as  these  with  a  chance  of  but  1  in  16  that 
any  single  consecutive  order  was  incorrect. 

It  will  be  observed  that  the  p.e.'s  obtained  in  the  study  by  Cattell 
are  somewhat  larger  than  those  here  presented.  One  cause  appears 
at  first  glance,  the  twenty  judgments  of  the  authors  as  against  the  ten 
of  the  men  of  science.  But  the  p.e.'s  of  ten  random  selections  are  also 
smaller  as  is  also  the  m.v.  Only  the  first  ten  Astronomers,  Anthro- 
pologists, Physiologists  and  Psychologists  have  p.e.'s  that  approach 
in  smallness  those  of  the  literary  men.  This  cannot  be  wholly  ascribed 
to  the  limitations  of  position  in  the  lower  grades.  To  what  extent 
differences  in  the  selection  of  the  groups  can  be  held  to  account  for 
the  disparity  may  also  be  questioned.  If  we  took  the  whole  thousand 
American  men  of  science  as  one  group  we  do  not  know  whether  the 
differences  in  the  first  ten  would  be  larger  or  smaller  than  in  the  first 
ten  authors.  It  is  true  that  there  are  always  more  writers  than  men 
of  science,  but  abler  men  may  be  drawn  to  the  sciences  and  especially 
would  this  be  the  case  near  the  top,  though  it  is  improbable  that  the 
psychological  limit  of  worthlessness  is  so  low  in  science  as  in  litera- 
ture. Opportunity  probably  counts  for  less  in  letters  than  in  science, 
and  the  literary  writer  seems  to  be  a  more  specialized  type.  Then 
too,  in  the  course  of  classifying  the  men  of  science  into  twelve  groups,  we 
might  find  that  the  differences  at  the  top  of  each  group  were  smaller 
than  at  the  top  of  the  total  of  the  groups.  It  is  hardly  possible  to  say 
whether  the  fact  that  only  living  men  are  included  should  make  the 
differences  smaller  or  larger. 

A  priori,  we  should  perhaps  expect  that,  with  equal  differences, 
the  grading  would  be  easier  for  the  scientific  than  for  the  literary  men 
on  account  of  the  greater  objectivity  of  scientific  work,  and  be- 
cause the  graders  were  selected  with  special  reference  to  their  knowl- 
edge of  this  work.  We  might  expect  individual  taste  to  effect  greater 
variability  in  the  authors.  But  it  is  the  natural  reply  that  the  literary 
graders  were  trained  in  making  just  this  sort  of  judgments,  and  that 
all  the  training  that  they  received  made  directly  for  greater  unanimity. 
This  is,  of  course,  a  disturbing  factor,  but  its  importance  could  be 
easily  exaggerated.  For  example,  it  may  be  questioned  whether 
the  graders  had  received  special  training  in  judging  the  relative  merits 
of  VII  and  IX  or  the  relative  euphoniousness  of  IV  and  V,  though 
there  is  no  abnormal  disagreement  in  either  case.     Previous  training 


ORDER,  POSITIONS,  ETC.  19 

doubtless  contributed  to  IPs  first  place  in  Originality,  but  it  will  be 
noted  that  the  p.e.'s  are  not  necessarily  smaller  where  previous  train- 
ing would  naturally  be  supposed  to  have  made  for  most  vmanimity 
of  judgment.  In  fact  there  seems  to  be  little  reason  why  these  judg- 
ments should  not  be  regarded  as  equally  naive  with  those  of  the 
men  of  science. 

To  compare  the  literary  with  the  scientific  p.e.'s  on  the  basis 
of  Cattell's  Table  IV  would  be  a  very  hazardous  task,  in  view  of  the  ad- 
mittedly unsettled  character  of  the  hypothesis  upon  which  this  ar- 
rangement is  based,  i.  €.,  that  the  range  of  ability  is  the  same  in  each 
science.  If  the  upper  ten  men  of  science  were  graded  by  judges  with 
proportioned  knowledge  of  their  work  the  first  three  would  hardly  have 
p.e.'s  of  .0,  as  it  is  of  course  necessary  to  assign  to  them  here;  the  re- 
maining p.e.'s  would  necessarily  be  much  smaller,  but  it  is  imprac- 
ticable even  to  guess  at  their  relation  to  the  literary  p.e.'s. 


11.  QUALITY  ANALYSIS. 

If  our  list  of  literan'  qualities  were  entire,  and  offered  a  com- 
plete analysis  of  all  kinds  of  literary  merit,  the  sum  of  the  grades 
of  an  author's  qualities,  properly  weighted,  should  give  an  exact  cor- 
respondence to  his  grades  in  general  merit.  It  is  of  course  imprac- 
ticable to  approach  the  problem  in  this  way,  it  being  attempted  merely 
to  cover  the  field  as  well  as  possible  with  ten  qualities.  How  well 
they  cover  the  field  of  general  merit  is  measured  by  the  degree  of  their 
correspondence  with  the  direct  grades  in  general  merit.  The  list  may 
also  cover  one  author's  qualities  more  completely  than  another's; 
in  this  case  the  former's  grades  would  approximate,  the  latter's  would 
diverge  from  the  general  merit  grade.  If  there  had  been  omitted  from 
the  list  some  important  quality  in  which  an  author  stood  well,  his 
grade  in  general  merit  would  be  higher  than  the  sum  of  his  grades 
in  the  qualities.  If  it  were  one  in  which  he  stood  poorly,  this  sum 
would  be  unfairly  advantageous  to  him.  On  account  of  the  fact 
that  the  relative  difference  in  the  importance  of  the  qualities  is  so 
great,  and  that  an  inordinately  high  or  low  grade  in  certain  qualities 
may  fall  to  a  poor  or  good  author,  the  median  is  a  better  measure 
in  this  case  than  the  average,  because  it  tends  to  automatically  weight 
the  significance  of  the  qualities.  As  before,  there  is  no  essential  differ- 
ence between  the  two,  but  the  median  should  give  in  general  a  fairer 
representation  of  the  truth  (see  General  Table,  cols.  M.  of  M.  and 
Av.  of  Av.). 

With  two  exceptions  to  the  median  and  three  to  the  average 
order  the  correspondence  is  complete.  Ill  receives  a  much  lower 
grade,  IV  a  slightly  lower  grade  in  the  median  of  their  qualities  than 
should  be  the  case.  A  very  satisfactory  analysis  of  the  other  authors 
is  afforded,  but  the  omission  of  certain  qualities  has  done  III  and  IV 
an  injustice.  In  the  case  of  III  this  was  anticipated  in  the  arrangement 
of  the  experiment.  The  intellectual  appeal  plays  but  a  minor  part  in 
the  list  of  qualities,  and  it  is  precisely  here  that  III  is  generally  sup- 
posed to  be  supreme.  It  is  probable  that  the  lesser  displacement  of 
IV  is  also  due  to  this  cause.  The  list  could  perhaps  be  improved  by 
substituting  for  one  of  the  qualities  something  that  would  cover  the 
intellectual  appeal.  In  view  of  the  results  obtained,  the  number  of 
qualities  ought  hardly  to  be  increased.  Ten  seems  to  cover  the  situa- 
tion as  completely  as  is  necessary.  Whether  this  would  be  the  case 
in  more  complicated  work,  as  with  human  character  and  tempera- 
ment, is  not  determined.  The  published  character  analysis  blanks 
generally  contain  a  much  larger  number  than  this.  Personally,  I 
am  inclined  to  think  that  ten  would  suffice.     For  practical  purposes 


QUALITY  ANALYSIS  21 

this  problem  would  not  be  quite  so  complex.  We  should  rather  wish 
to  know  a  person's  standing  in  a  certain  quality  for  itself,  irrespec- 
tive of  its  relation  to  the  general  complex  of  character. 

It  is  possible  by  various  devices  to  measure  the  degree  of  corre- 
spondence between  the  judgments  in  general  merit  and  those  in  the 
qualities.  Some  qualities  are  found  to  depart  almost  twice  as  much 
as  others  from  the  general  determinations.  In  an  entirely  empirical 
sense,  this  degree  of  correspondence  may  be  interpreted  as  furnishing 
a  measure  of  the  relative  importance  of  any  of  the  given  qualities  in 
determining  the  author's  position  in  general  merit.  From  the  present 
data  it  would  probably  be  unjust  to  infer  that  any  of  the  qualities 
named  was  an  active  disadvantage  to  an  author,  or  that  there  were 
likely  to  be  any  striking  correlations  between  the  different  qualities 
themselves. 

While  the  results  of  the  determinations  appear  applicable  to  this 
particular  group  of  authors,  their  value  as  general  measures  of  rela- 
tive importance  would  depend  on  the  supposition  that  the  ranges  in 
the  various  qualities  were  somew^here  near  the  same.  There  is  i^erhaps 
no  particular  reason  why  they  should  be  the  same,  and  to  a  certain 
extent  differences  would  be  indicated  in  the  positions  and  p.e.'s  them- 
selves. For  example,  Charm  might  not  be  a  particularly  important 
trait,  yet  it  might  be  so  absolutely  preeminent  in  an  author  that  it 
raised  his  general  position  higher  than  it  should.  The  differences 
in  the  ranges  as  indicated  by  the  figures  given  do  not,  however,  seem 
to  be  such  as  to  render  the  calculations  less  worth  while.  And  such 
confidence  in  their  validity  as  might  be  derived  from  further  analysis 
of  the  results  themselves,  one  of  the  methods  at  least  does  not  fail 
to  give. 

There  is  a  possibility  of  one  rather  disturbing  constant  error  in 
measures  of  this  nature,  whose  extent  it  is  never  possible  to  know 
accurately.  There  is  noted  introspectively  a  tendency  to  grade  for 
general  merit  at  the  same  time  as  for  the  qualities,  and  to  allow  an 
individual's  general  position  to  influence  his  position  in  the  qualities. 
This  would  be  the  case  especially  in  the  case  of  those  qualities  that 
were  ill-defined  in  the  minds  of  the  subjects,  and  tended  to  l)e  inter- 
preted rather  in  terms  of  general  merit.  We  might  thus  have  a  grad- 
ing of  Charm  by  general  merit  instead  of  general  merit  by  Charm. 
This  would  make  the  correspondences  of  such  qualities  api^ear  closer 
than  they  were.  It  probably  does  not  play  any  serious  part  save  i>er- 
haps  with  Proportion.  It  may  also  contribute  to  the  high  position 
of  Finish,  but  it  is  difficult  to  see  how  it  could  have  been  avoided. 

The  results  of  the  calculations  by  the  various  methods  to  be 
described  are  given  in  the  accompanying  table. 


22         .4  STATISTICAL  STUDY  OF  LITERARY  MERIT 


A.     In 

d.  Dis. 

B.  Med.  Dis. 

C.  Rel.  to  Med. 
of  Med.  and 
Med.  of  G.M. 

D.  Med.  ft 
like  signs. 

K.  Size 
of  p.e.' 

s. 

Fi. 

Ord. 

Pos.        P 

.E. 

Ord. 

Pos. 

Ord. 

Pos. 

Ord. 

Pos. 

Ord. 

Pos. 

I. 

12.2    1 

7 

II. 

6 

I. 

7 

I. 

8 

I. 

.307 

Eu. 

II. 

12 

7 

7 

I. 

7 

II. 

7 

II. 

8 

II. 

312 

Or. 

III. 

13 

4 

6 

IV. 

lO 

III. 

7 

V. 

8 

IV. 

.320 

Im. 

IV. 

14 

9 

V. 

II 

IV. 

6 

VI. 

8 

III. 

322 

Pr. 

V. 

14 

6 

8 

III. 

13 

V. 

6 

IX. 

7 

VI. 

328 

Fo. 

VI. 

14 

7 

8 

VI. 

14 

VIII. 

6 

III. 

6 

VII. 

363 

Ch. 

VII. 

i.S 

.s 

8 

VII. 

17 

VI. 

4 

IV. 

6 

V. 

365 

Sy. 

VIII. 

iq 

6 

8 

IX. 

18 

VII. 

4 

VII. 

6 

VIII. 

369 

Wh. 

IX. 

20 

Q 

8 

VIII. 

19 

IX. 

2 

VIII. 

5 

IX. 

376 

CI. 

X. 

22 

7     I 

o 

X. 

20 

X. 

I 

X. 

4 

X. 

413 

Thus  the  average  number  of  displacements  per  individual  is  12.7  in  Eu.,  15.5  in  Ch., 
etc.  but  the  number  of  displacements  for  the  median  grades  of  the  group  is  7  for 
Fi.,  18  for  Wh.,  etc. 

The  general  average  of  the  individual  displacements  is  16. i  with   an  m.v.  of   5.1, 
the  distribution  of  the  entire  200  series  of  displacements  being  as  follows: 
4     6     8     10     12     14     16     18     20     22     24     26     28     30     38 
7     7   17     21      19     30     20     25     14     14       7       9       5       4       I 
The  distribution  is  again  skewed  to  the  small  end,  like  that  of  the  p.e.'s.  and  prob- 
ably for  the  same  reason,  i.  e.,  hmit  of  individual  accordance. 

A  rough  determination  of  the  standards  by  which  our  20  graders 
judged  as  a  group  may  be  rapidly  arrived  at  by  simply  making  a  table 
in  which  a  +  sign  is  attached  to  every  case  in  which  the  quality  grade 
of  an  author  is  on  the  same  side  of  the  median  of  the  grades  in  that 
quality  as  the  author's  grade  is  on  the  side  of  the  median  of  general 
merit.  A  —  sign  means  that  the  quality  grade  and  the  grade  in  gen- 
eral merit  are  on  different  sides  of  their  respective  medians.  Thus  I  in 
general  merit  is  also  high  in  Charm,  and  for  this  quality  receives  a  + 
sign.  But  he  is  low  in  Wholesomeness,  and  in  this  receives  a  —  sign. 
Then  the  quality  in  which  the  greatest  number  of  +  signs  is  found  is 
that  quality  in  which  an  author  oftenest  stands  in  a  position  analogous 
to  his  place  in  general  merit.  As  will  be  seen,  high  and  low  positions 
in  general  merit  have  usually  gone  with  high  and  low  positions  in 
Euphony,  Finish,  and  Imagination,  but  only  once  has  this  been  the 
case  in  Clearness   (Table,  col.  C). 

Correlations  by  %  of  like  signs  were  applied,  but  the  results  were 
very  inferior  to  those  obtained  by  the  other  methods,  as  shovm  in 
column  D.  It  shows  just  enough  agreement  to  demonstrate  its  in- 
exactness. While  well  adapted  for  certain  sorts  of  work  and  the  only 
method  for  cursory  observation  of  individual  relationships,  it  does 
not  seem  to  operate  satisfactorily  in  the  correlation  of  orders. 

It  would  be  difficult,  however,  to  find  a  correlation  method  more 


QUALITY  ANALYSIS  23 

admirably  adapted  to  all  relative  position  work  than  the  measure  of 
displacements  devised  by  Professor  Woodworth.  In  any  order  of 
10  positions,  such  as  we  have  here,  to  produce  an  exactly  reverse 
order  {i.  e.,  correlation  — 100%  Pearson)  would  require  45  displace- 
ments. X  being  above  9  that  he  should  be  Ijelow  gives  9  displace- 
ments, IX  above  8  that  he  should  be  below  gives  8,  etc.,  total  45. 
Orders  that  had  no  reference  to  the  standard  would  center  about  22 
and  23  displacements,  while  the  fewer  the  displacements  the  higher 
the  positive  correlation.  For  comparative  purposes  the  displace- 
ments may  be  expressed  in  percentile  relation. 

There  have  been  determined  by  this  method  the  number  of  dis- 
placements from  the  order  of  general  merit  given  by  the  order  in  each 
of  the  qualities  (see  Table,  col.  B).  This  is  a  rapid  means  of  reaching 
a  generally  reliable  conclusion,  and  is  much  more  exact  than  that  af- 
forded by  the  relation  of  the  individual  positions  to  the  general  median. 
It  is  as  yet  impracticable,  however,  to  assign  a  workable  p.e.  in  such 
determinations  and  for  this  purpose  I  undertook  the  calculation  of 
the  displacements  of  each  quality  as  given  by  each  individual  grader 
from  the  order  of  general  merit  as  given  by  that  individual.  The 
order  of  correspondence  thus  obtained  has  been  taken  as  the  standard 
(col.  A),  as  it  seems  to  possess  a  measurable  and  not  inconsiderable 
degree  of  validity.  According  to  the  graphic  representation  the  jDosi- 
tions  and  p.e.'s  are  as  follows: 


/i    i3    If    ir  /I    /7    1*    /I    >«    w    >i    «    ;» 

The  p.e.'s  of  the  average  displacements  are  larger,  yet  the  differ- 
ences are  usually  distinct  within  two  places.  The  steps  are  about 
equal  for  the  first  seven  qualities,  and  then  we  find  a  considerable 
gap  to  the  last  three,  whose  p.e.'s  are  larger,  as  those  at  the  top  are 
smaller.  Some  traces  of  this  gap  are  discernible  in  the  results  by 
the  cruder  methods.  Indeed  not  the  least  reason  for  confidence  in  these 
orders  is  the  correspondence  they  maintain.  The  B  and  C  orders  are 
practically  the  same  while  the  very  coarsely  determined  D  order 
keeps  well  on  the  positive  side.  The  siun  of  these  orders  is  prac- 
tically that  given  by  the  standard. 

The  above  orders  are  all  measures  of  the  same  general  thing, 
between  which,  provided  they  were  valid  in  principle,  a  certain  cor- 
respondence would  be  mathematically  necessary.  A  still  closer  cor- 
respondence, however,  is  found  with  an  order  mathematically  by  no 
means  so  well  associated  with  the  degree  of  correspondence,  namely, 
the  size  of  the  p.e.'s  discussed  on  p.  16,  and  whose  table  is  reproduced 
in  col.  E.     It  will  be  noted  that  the  order  of  relative  importance  of 


24 


A  STATISTICAL  STUDY  OF  LITERARY  MERIT 


the  qualities  corresponds  to  the  order  in  size  of  the  average  p.e.  with 
but  three  displacements.  A  certain  amount  of  this  must  indeed  be 
ascribed  to  happy  chance,  for  the  differences  in  the  p.e.'s  are  often 
infinitesimal,  and  were  there  actuallj''  perfect  correspondence  the  pres- 
ent methods  would  be  far  too  coarse  to  detect  it  surely.  So  far  as  the 
results  go,  the  qualities  that  we  tend  to  judge  an  author  by  are  also 
those  that  we  tend  to  grade  with  the  greater  accuracy.  It  is  perhaps 
not  unnatural  that  the  traits  about  which  we  have  the  most  assurance 
should  also  be  those  that  we  regard  as  the  most  important.  The 
close  correspondence  of  the  two  may  itself  be  in  the  nature  of  an  argu- 
ment for  their  validity. 

The  method  measures  directly  an  author's  possession  of  a  quality 
with  reference  to  other  authors.  Indirectly  an  idea  may  be  obtained 
of  the  prominence  or  absence  of  a  quality  relative  to  the  other  quali- 
ties of  his  own  work.  Aside  from  such  errors  as  would  be  due  to 
differences  in  the  ranges,  etc.,  he  is  likely  to  have  more  of  a  quality 
in  which  his  position  is  higher  than  of  one  in  which  his  position  is 
lower.  Thus  I,  who  has  a  median  of  2.1  in  Imagination,  but  one  of 
6.9  in  Wholesomeness,  is  probably  more  imaginative  than  he  is  whole- 
some. A  table  may  be  constructed  in  which  a  plus  sign  is  given  to  those 
quality  grades  which  are  at  the  same  time  both  above  the  author's 
median  of  medians  and  the  general  median  of  the  grades  in  that  quality, 
this  last  always  falling  somewhere  in  the  neighborhood  of  5.5.  Minus 
is  assigned  to  those  grades  which  fall  at  the  same  time  below  the  author's 
median  of  medians  and  the  general  median  of  the  quality,  and  a 
zero  sign  goes  to  those  which  fall  between  the  two.  Other  things 
being  equal,  a  +  sign  then  goes  to  the  qualities  that  are  relatively 
prominent,  a  —  sign  to  those  that  are  absent,  and  zero  to  those  which 
are  inconspicuous  one  way  or  the  other.  Such  a  table  contains  35 
+  signs,  27  —  signs,  and  38  zero  signs.  The  figure,  however,  has 
little  significance  save  when  it  refers  to  a  prominent  quality  in  a  low 
author  or  a  lacking  quality  in  a  high  one.  The  following  are  in  order 
the  two  highest  and  the  two  lowest  quality  grades  received  by  each 
author;  i.  e.,  the  two  qualities  for  which  his  work  is  presumably  the 
most  and  the  least  distinguished. 


Most. 

lycast. 

Most. 

Least. 

I. 

Fi 

Im 

CI 

Wh 

VI. 

Ch 

CI 

Or 

Fo 

II. 

Im 

Eu 

Sy 

Wh 

VII. 

Wh 

Eu 

Or 

Ch 

III. 

Fo 

Wh 

Pr 

CI 

VIII. 

Or 

Fo 

Fi 

Pr 

IV. 

Eu 

Pr 

Im 

Or 

IX. 

Ch 

CI 

Fo 

Im 

V. 

Sy 

Wh 

Fo 

Or 

X. 

Im 

CI 

Eu 

Fi 

^^^L^  ^c^.^^jut^  U^  JU^  U^   iL-A^  :  lu^su  t1  ^'^^^ 


»''V'i--mA 


t)  Guci.'^(>^ci.s.  y^   ^1S- 


III.  ON    THE    VALIDITY    OF    INDIVIDUAL    JUDGMENT    AS 
MEASURED  BY  DEPARTURE  FROM  AN  ANERAGE. 

If  we  took  a  series  of  graduated  weights,  and  asked  a  numljer 
of  persons  to  serially  arrange  them  in  order  of  their  apparent  heavi- 
ness, we  should  find,  if  the  differences  between  the  weights  were  suffi- 
ciently small,  that  no  one  could  save  by  chance  arrange  them  in  cor- 
rect order,  but  that  there  would  always  be  more  or  less  displacement. 
The  person  whose  arrangement  showed  the  least  displacement  would 
approximate  closest  to  the  true  order,  and  we  should  therefore  con- 
sider him  to  have  the  most  accurate  judgment  for  weight.  Now 
assuming  that  the  distribution  of  all  the  errors  made  followed  that 
of  the  probability  curve,  we  should  find  that  the  errors  compensated 
and  that  the  average  order  in  which  the  weights  were  placed  would 
also  be  very  close  to  the  correct  order,  closer  probably  than  that 
of  the  best  individual,  though  the  average  number  of  displacements 
might  be  considerable.  In  estimating  the  accuracy  of  our  subjects' 
judgments  of  weight,  it  would  make  little  or  no  difference  whether  we 
took  as  the  true  order  the  actual  order  of  heaviness  as  measured  on 
the  scales,  or  took  the  average  order  as  the  standard.  Theoretically, 
each  would  give  us  the  same  result. 

But  there  are  many  important  qualities,  and  indeed  those  most 
adaptable  to  measurement  by  relative  position,  whose  differences 
we  cannot  determine  in  this  objective  way.  The  question  then  arises, 
are  we  also  here  justified  in  taking  the  truth  of  the  average  order  as 
objective,  and  measuring  the  value  of  a  judgment  according  to  its 
deviation  from  it?  For  clearly  unless  our  average  approximates 
to  some  objective  validity,  the  absolute  value  of  a  single  judgment  is 
not  measured  l^y  the  amount  of  its  deviation  from  it.  To  recur  to 
our  weights,  suppose  we  heated  and  cooled  the  weights  to  varying 
degrees  before  presenting  them  to  all  save  one  of  our  subjects,  and 
to  him  presented  them  at  equal  temperatures.  The  subjects  would 
all  feel  the  colder  weights  as  heavier,  and  the  average  order  would 
not  be  the  objectively  true  one,  and  the  order  of  the  subject  jx^rceiv- 
ing  the  weights  under  equal  conditions  might  well  be  the  farthest 
from  the  average.  Our  two  groups  would  give  us  different  results 
because  they  were  judging  from  different  standards. 

It  is  just  this  condition  that  must  be  guarded  against  in  those 
measurements  where  an  average  order  is  all  that  we  have  to  guide 
us.  We  have,  a  priori,  no  objective  measure  of  the  var^'ing  stand- 
ards by  which  the  individuals  judge.  Still  less  do  we  know  the  rela- 
ive  values  of  the  standards  themselves.     In  the  case  of  the  weights 


26         A  STATISTICAL  STUDY  OF  LITERARY  MERIT 

we  know  the  differing  nature  of  the  standards,  and  can  allow  for 
them;  but  if  we  did  not  know  them  the  judgment  of  the  single  subject 
would  still  be  the  most  useful  for  us.  Practise  will  overcome  many 
illusory  standards  of  judgment  to  which  normal  persons  are  subject, 
and  I  should  hardly  have  the  right  to  assert  my  judgment  of  direc- 
tion to  be  superior  to  that  of  Professor  Judd  because  I  was  nearer 
tlie  average  than  he  in  amount  of  subjection  to  the  Zollner  illusion. 

In  the  measurement  of  mental  traits  by  relative  position  we 
have  thus  two  factors  that  tend  to  cause  individual  deviation  from 
the  average,  namely  the  absolute  inaccuracy  of  the  judgment,  the 
direction  of  whose  errors  will  be  variable,  and  a  differing  standard 
from  other  members  of  the  group,  the  direction  of  whose  errors  will 
be  constant,  at  least  throughout  the  individual.  We  must  know  the 
exact  nature  of  the  deviations  due  to  these  two  causes  before  we  can 
estimate  the  values  of  the  judgments.  We  must  also  know  the  value 
of  the  standards,  for  it  is  possible  that  the  opinion  of  a  very  accurate 
judge  by  one  set  of  standards  might  be  of  smaller  value  than  that  of 
a  less  accurate  judge  by  another.  We  must  show  cause  why  a  person 
who  judges  literary  work  by  its  clearness  must  have  ipso  facto  a  poorer 
judgment  than  one  who  judges  it  by  its  imagination. 

It  is  possible  that  in  the  estimation  of  scientific  merit,  where 
this  method  found  its  first  application,  there  would  be  more  unanim- 
ity in  the  standards  of  judgment,  yet  there  are  some  divergences 
from  this  cause,  since  there  was  an  observed  tendency  for  graders 
to  give  disproportionately  high  position  to  men  engaged  in  the  same 
special  work  with  them  and  to  their  own  immediate  colleagues.  The 
method  has  here  been  applied  only  to  the  first  fifty  psychologists, 
but  it  gave  fairly  definite  results,  and  these  might  be  still  more  definite 
in  others  of  the  sciences.  Save  for  observer  A  the  order  is  rather 
variable,  and  it  might  be  questioned  whether  a  man's  estimate  of  the 
fifth  group  should  be  allowed  the  same  weight  with  his  estimate  of 
the  first.  This  is  also  a  matter  subject  to  a  good  deal  of  variation, 
for  the  second  best  judge  of  the  first  ten  psychologists  is  the  worst 
of  the  second,  the  fifth  of  the  third,  the  eighth  of  the  fourth,  and  the 
sixth  of  the  fifth. 

However,  where  the  variations  in  the  standards  compensate,. 
as  they  ought  to  do  in  scientific  merit,  the  method  is  immeasurably 
more  valid  than  where  they  not  only  patently  fail  to  do  so  but  give 
a  false  standard,  as  in  literary  merit.  The  conditions  are  exactly 
the  same  as  with  the  varying  sizes  and  temperatures  of  the  weights^ 
Our  group  of  weight-graders  constantly  gives  a  small  or  cold  object 
an  undue  weight;  the  group  of  scientific  graders  constantly  assigns 
high  position  to  their  immediate  colleagues  and  co-workers;  the  group 
of  literary  graders  constantly  allows  a  presumably  undue  weight  to 


VALIDITY  OF  IXDIVIDUA L  JlJUdMESr  TI 

Euphony  and  Finish.  The  variation  in  the  accordance  of  the  judges 
is  a  little  over  2:1,  as  was  the  case  in  Cattell's  psychologists;  the  ac- 
cordance of  the  judgments  also  tends  to  follow  the  normal  distribution, 
though  there  seems  to  be  a  slight  skew  in  favor  of  the  more  accordant 
judgments. 

It  should  not  be  iinpossible  to  get  a  quantitative  demonstration 
of  these  differing  standards.  When  we  have  a  series  of  objects  graded 
in  respect  to  a  general  quality,  and  then  in  regard  to  the  main  ele- 
ments of  that  quality,  the  relative  influence  of  the  elements  on  the 
general  judgment  appears  in  their  degree  of  correspondence  to  the 
general  ciuality.  Now  while  the  graders  showed  a  certain  unanimity 
in  assigning  to  various  elements  of  literary  merit  a  certain  order  of 
influence,  it  does  not  follow  that  the  mature  judgment  of  eminent 
literary  critics  would  give  the  same  order,  or  that  the  graders  them- 
selves would  give  it  twenty  years  hence.  Still  less  does  it  follow 
that  this  standard  is  the  best  one  for  us  to  abide  by,  or  that  it  is  one 
which  the  graders  themselves  would  not  be  among  the  first  to  con- 
sciously repudiate.  If  we  had  the  qualities  directly  graded  in  order 
of  value  to  literary  merit,  we  should  hardly  expect  to  find  Euphony 
and  Finish  first,  Clearness  and  Wholesomeness  last.     Nor  do  we. 

Such  a  judgment  was  obtained  from  a  group  of  24  graduates  in 
psychology  and  education,  of  about  the  same  intellectual  level  as  those 
who  furnished  the  literary  grades.  I  see  no  reason  a  priori — and 
there  is  certainly  none  evident  in  the  results — why  the  conscious 
judgment  of  this  group  should  not  have  the  same  ethical  value  as  that 
of  the  literary  graders,  or  why  the  terms  should  not  have  been  equally 
well  understood.  The  group  contained  a  certain^  proportion  of 
women,  about  one-third,  but  this  factor  did  not  appear  to  influence 
the  character  of  the  judgments.  The  formula  by  which  the  ciuali- 
ties  were  graded  was  "according  to  their  importance  to  the  fulfil- 
ment of  the  highest  function  of  literature."  No  definitions  of  any 
of  the  qualities  were  given,  nor  does  it  appear  that  it  would  have  ijeen 
advantageous  to  have  given  them.  This  order  of  importance,  with 
positions  and  p.e.'s,  is  shown  in  the  accompanying  table  (cols.  T.C.). 

This  table,  compared  with  that  on  p.  22,  gives  an  idea  of  what 
we  think  we  judge  literary  merit  by  as  contrasted  with  what  we  ac- 
tually judge  it  by.  The  number  of  displacements  between  the  two 
orders  is  28 — slightly  more  than  we  should  expect  by  pure  chance. 
Such  correspondence  as  there  is  between  our  naive  and  conscious 
standards  is  thus  slightly  in  the  direction  of  perversity.  It  is  proba- 
bly something  more  than  an  amusing  coincitlence  that  that  quality 
which  we  are  so  sure  we  ought  to  judge  an  author  by  most  of  all  is 
the  one  Avhich  really  plays  the  least  part  in  our  estimate  of  liim.  ami 
that  the  two  qualities  which  ought  to  have  the  least  share  in  deter- 


28 


A  STATISTICAL  STUDY  OF  LITERARY  MERIT 


T.  C. 

E.  G.    T.  C 

E 

.  G. 

Av.       1 

p.  e. 

Cl. 

2-7 

4.1 

28 

43 

Fo. 

3 

7 

4 

7 

26 

41 

Or 

3 

7 

3 

37 

31 

Im. 

4 

3 

2 

4 

35 

34 

Wh. 

5 

5 

6 

9 

49 

6 

Pr. 

6 

I 

6 

3 

25 

55 

Ch. 

6 

I 

5 

8 

33 

55 

Sy. 

6 

3 

5 

7 

31 

46 

Fi. 

8 

4 

7 

3 

32 

21 

Eu. 

3 

5 

9 

I 

i8 

26 

A.D.P. 

!    -^5 

75  Av. 

•3 

39 

mining  an  author's  position  are  those  which  always  show  the  most 
remarkable  correspondence  with  it. 

The  distributions  of  these  grades  are  unimodal  for  the  most 
part,  and  only  in  Wholesomeness  do  we  find  distinct  species  of  high 
and  low  grades.  It  has  much  the  largest  p.e.  and  is  the  only  quality 
receiving  a  grade  in  every  place.  The  species  were  examined  for 
sex  correlations,  but  none  were  apparent. 

Before  the  method  for  the  determination  of  individual  standards 
had  been  applied,  the  literary  graders  had  been  made  aware,  through 
one  of  the  cruder  methods,  of  the  general  relations  of  the  qualities.  It 
was  therefore  impossible  to  obtain  from  them  any  order  not  subject 
to  large  constant  error.  Nevertheless,  it  seemed  worth  while  to  ob- 
tain a  few  records  from  this  group. 

Records  were  obtained  from  14  individuals,  of  whom  12  had 
taken  part  in  the  previous  test.  The  results  are  given  in  the  last 
quoted  table,  cols.  E.G.  The  order  and  positions  here  assigned 
also  differ  from  the  objectively  determined  order  by  slightly  more 
than  the  chance  nimiber  of  displacements,  but  while  the  number  of 
displacements  is  almost  identical  with  that  of  the  order  given  by  the 
other  group,  there  are  11  displacements  between  the  two  groups 
themselves,  and  in  a  few  cases  these  discrepancies  are  outside  the 
limits  of  the  p.e.  This  may  well  be  due  to  the  constant  error  men- 
tioned above,  and  I  do  not  consider  that  there  is  sufficient  warrant 
for  supposing  separate  species.  An  interesting  aspect  of  these  re- 
sults is  afforded  from  the  view-point  of  individual  comparisons.  The 
number  of  displacements  that  occur  between  the  order  of  the  authors 
in  general  merit  and  their  order  as  assigned  in  the  various  qualities 
by  a  single  individual,  gives  an  idea  of  that  individual's  actual  stand- 
ards of  judgment.  The  qualities  that  vary  least  from  the  general 
merit  order  are  his  most  important  standards.  In  the  grading  of 
the  qualities  themselves  we  have  the  conscious  standards  by  which 


VALIDITY  OF  INDIVIDUAL  JUDGMENT  29 

the  individual  thinks  he  judges.  The  orders  assigned  to  the  quaH- 
ties  naively  and  consciously  are  strikingly  divergent.  The  average 
number  of  displacements  is  about  20,  a  little  less  than  the  chance 
number;  it  occurs  as  high  as  34,  and  as  low  as  8.  In  the  former  case 
the  individual's  conscious  standards  are  almost  the  reverse  of  his 
naive  standards.  We  might  call  such  a  figure  a  "coefficient  of  con- 
sistency," 

The  relative  smallness  of  the  p.e.'s  of  the  averages  assigned  by 
the  Teachers'  College  Group  is  due  wholly  to  the  larger  number  of 
graders;  the  p.e.  of  the  individual  judgment,  as  measured  by  the  m. v., 
is  practically  the  same  in  each  group.  It  is  interesting  to  observe 
that  the  special  training  of  the  literary  graders  has  neither  varied 
the  standards  to  any  noteworthy  degree,  nor  given  them  greater  as- 
surance. 

There  are  many  complications  into  which  it  is  not  possible  to 
enter  deeply.  Thus  a  certain  irreducible  minimum  of  Clearness  might 
be  most  desirable,  but  once  this  irreducible  minimum  were  assumed, 
an  analogous  degree  of  Charm  might  be  more  important.  It  must 
also  be  remembered  that  the  standards  quoted  in  the  table  on  p.  22 
are  standards  for  the  criticism  of  imaginative  writers,  while  the  quali- 
ties are  here  graded  according  to  their  importance  to  the  fulfilment 
of  the  highest  function  of  literature.  If  we  had  graded  a  group  of 
historians,  we  should  probably  have  found  less  real  judging  by  Euphony 
and  Finish,  and  more  by  Clearness  and  Force.  The  standards  of 
judgment  for  imagmative  writing  may  not  be  the  highest  literary 
standards,  perhaps  there  are  other  departments  of  literature  which 
are  held  to  higher  standards.  But  this  interpretation  is  of  very 
doubtful  value,  since  literature,  technically  considered,  is  imagina- 
tive by  definition. 

Now  the  best  judge  is  not  the  man  who  judges  most  true  to  ordi- 
nary standards,  but  the  man  who  judges  most  true  to  the  best  stand- 
ards. To  discuss  what  these  best  standards  might  be  would  lead  at 
once  into  devious  ethical  pathways;  let  us  call  them  for  the  moment 
the  most  useful  ones.  It  is  probably  fair  to  assume  that  the  maturer, 
more  experienced  and  distinguished  of  a  group  of  graders,  selected 
by  universal  experience  for  the  very  abilities  which  they  are  here 
exercising,  shoukl,  at  least  in  this  particular  respect,  have  a  better 
judgment  than  the  remainder  of  the  graders.  By  this  same  token, 
they  should  also  have  different  standards  of  judgment,  and  this  would 
tend  to  draw  them  awa}^  from  the  average,  but  should  not,  therefore, 
be  held  to  discount  the  value  of  their  opinions.  After  all,  the  func- 
tion of  a  method  of  this  sort  is  not  to  tell  us  what  we  could  not  possi- 
bly find  out  in  any  other  way,  but  rather  to  determine  quickly  what 
in  less  organized  experience  might  require  many  years.     Its  data 


30  A  STATISTICAL  STUDY  OF  LITERARY  MERIT 

must  not  run  too  contrary  with  those  of  our  every-day  experience; 
even  the  method  of  measurement  by  relative  position  would  itself 
hardly  survive  the  shock  of  Aristotle's  appearing  in  the  lower  half 
of  the  world's  philosophers.  The  data  of  relative  critical  ability 
obtained  by  this  method  show  little  accordance  with  the  results  of 
oiu"  partially  organized  experience.  It  is  also  true  that  there  is  ap- 
parent in  the  results  no  correlation  between  accordance  of  judgment 
to  the  average  and  approximation  of  individual  standards  to  it;  how- 
ever, when  the  new  factors  that  would  here  come  into  play  are  con- 
sidered, it  will  easily  be  seen  that  the  present  data  are  much  too  coarse 
for  such  refinements.  But  the  order  of  critical  ability  given  by  the 
method  of  direct  accordance  is  quite  too  far  from  that  of  the  best  ex- 
perience. Nor  does  the  best  judgment  for  literary  merit  correspond 
at  all  to  the  best  judgment  for  the  various  qualities.  The  worst  judge 
of  general  literary  merit,  according  to  his  divergences,  is  the  3rd  best 
judge  of  Charm,  the  best  judge  of  Clearness,  and  the  13th  best  of 
Euphony.  The  best  judge  of  general  merit  is  the  5th  best  of  Charm, 
the  14th  of  Clearness,  and  the  17th  of  Euphony. 

All  that  is  really  given  in  the  individual  deviations  from  the  aver- 
age judgment  is  the  individual  who  tells  us  most  about  the  group,  or 
the  most  accurate  judge  for  a  certain  set  of  standards,  which,  at  least 
in  the  case  of  these  literary  judgments,  every  one  will  probably  admit 
to  have  a  rather  low  ethical  value. 

We  can  hardly  draw  inferences  as  to  the  general  capacity  for 
sound  judgment  as  measured  by  the  soundness  of  judgment  for  any 
particular  class  of  objects.  We  must  have  the  information  as  well 
as  the  ability  to  weight  it.  It  might  be  that  the  best  judge  of  the 
psychologists  was  he  who  had  the  best  proportioned  knowledge  of 
the  work  done  in  the  various  fields.  Judgment  may  be  wholly  a  mat- 
ter of  information  if  we  make  this  term  synonymous  with  experience. 
Obviously  then,  the  fact  that  one  has  a  good  judgment  for  psycholo- 
gists tells  us  very  little  about  the  value  of  his  opinion  in  other  fields. 
To  demonstrate  the  very  existence  of  an  abstract  power  of  judgment 
is  ultimately  synonymous  with  the  problem  of  free  will.  Fortunately 
it  is  not  in  this  abstract  power  of  judgment  that  we  need  be  in  the 
least  interested,  but  rather  in  the  quality  of  one's  judgment  for  a  par- 
ticular class  of  objects.  We  wish  to  know  whether  a  person  is  a  good 
judge  of  distance,  of  faces,  of  a  mining  prospect.  To  determine 
this  we  must  pay  careful  attention  to  the  weighting  of  the  standards 
of   judgment. 


^5  if/ 


UC  SOUTHERN  REGIONAL  LIBRARY  FACILITY 


AA    000  807  140    9 


